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ABSTRACT 

The Research and Development Exchange (RDx) is a 
network of regional educational laboratories and university-based 
Teseirch and development centers working to support state and local 
school improvement efforts* Initiated by the National Institute of 
Education in 197^, the RDx is composed of eight Regional Exchanges 
and four central support services* In 1979 members of the RDx agreed 
to collaborate in the development of a resource base on the 
identif icatioT. and validation of promising educational programs, 
practices, and products* This pap^r provides a brief history and 
description of the following validation processes: . the Joint 
Dissemination Review Panel (JDRP) : Identification, Validation, 
Dissemination (IVD); Sharing BusinesSs^ Success (SBS) ; Project 
Information Packages (PIPs) ; and the National Education Practices 
File. Some of the issues in validation discussed in the paper include 
replication versus adaptation, the evaluation experience of the 
project staff, state responsibilities before validation, the degree' 
of rigor applied in validation, and the nature of on-site, visits* A 
bibliography on validation processes and the abbreviated form of 
original IVD submission requirements are ajjpended. (Author/MLF) 
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The Search fpr'Quallty Control in 
Dissemination of Educational 'Products and Practices: 
A Look at the Literature and Major Issues 



Linda Reed ^ 
R&O Interpretation Service 

INTRODUCTION 

The~concern about validation of educational programs and practices— a 
relatively new one for the educational world— has been a phenomenon of the 
seventies. Four early Influences creating this phenomenon were (1) the need 
to Justify the substantial amounts of federal monies that had been pouring 
Into local education agencies which would not continue to be available In such 
sizeable sums, (2) the need to justify the expenditures by demonstrating that 
the outcomes of federally funded LEA programs are general Izable, (3) the need 
to attest to the reliability of the programs and practices to be disseminated, 
and (4) the limitations on the ability of such national systems as ERIC to ^ 
effectively dissemination "exemplary* programs and practices. 

IrTresponse to the first three concerns, the United States Office of Education 
sponsored a number of early efforts to build quality control Into the 
identification and review of effective programs and products. These early 
attempts led ultimately to the development of. two separate processes: the 
Dissemination Review Panel (ORP) process ^and the Identification, Validation, 
dissemination (IVO) process. Both were put Into operation In 1972 for 
essentially the same purpose: 



to establish credible standards and procedures for identifying 
successful prograns, to provide for consumer protection when user 
school districts are searching for successful programs wttlch will 
help them solve educational problems or meet identified needs and 
to prevent the continued use of scarce funds in "reinventing the 
wheel."' 

> 

In 1979, after two years of pilot testing, a procedure for validating 
successful school business practices was published by the Research Corporation 
of the Association of School Business Officials in cooperation with the United 
States Office of Education and the State Departments of Education. The 
Sharing Business Success handbook outlines validation procedures for such 
practices as budgeting and financial planning, data processing, negotiations, 
office management, operation and maintenance of plants, personnel management, 
.pUnt ^planning and-Ci)nstnjct1on, professional development, property 
management, pupil transportation, purchasing and supply management, school 
fbod snd nutrition management, school safety, and energy conversation. 

Other efforts which wll I be discussed In this paper Include the Project 
Information Packages (PIPs) Project and the Bibliographic .Retrieval Service's 
(BRS) National Education Practices File. 

OVERVIEW OF MAJOR VALIDATION PROCEDURES 

The Joint Dissemination Review Panel 

Approximately 250 programs have been validated since 1972 by the Joint 
Dissemination Review'Panel (JDRP), originally the Dissemination Review Panel. 
In 1975 the review panel was enlarged by the addition of representatives frojn 
the National Institute of Education (NIE) and assumed its new title. From 
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1975 to 1980, the panel was composed of 22 members, eleven from USOE, 
appointed by the Commissioner of Education, and eleven from NIE, appointed by 
the Director of the Institute. Members of the panel were chosen for their 
experience In education and for the|r ability to analyze evaluative evidence 
on the effectiveness of educational products and practices. Although whe 
advent of the Department of Education and, more recently, the Republican 

I 

administration 9 has led to no Immediate changes In the composition of the 
panel. It Is possible that there win be modifications In the coming year or 
two* 

« 

JORP review was originally confined to programs developed with federal funds. 
The panel now reviews a broad range of programs which come from all states and 
which have been developed with funds from a variety of sources. Only" 
proprietary projects and products are not eligible for review. 

9 

2 

The Joint Oissemination Review Panel Ideabook , the major publication of the 
panel, establishes, and explains in detail, criteria for evaluating programs, 
giving samples of convincing and non-convincing data from validation 
applicants. The Ideabook also lists typical evaluation hazards which 
validation- teams should be alert for. 

Six questions must be addressed by a validation team as it examines an 
educational practice, although the Ideabook makes it clear that evidence of 
"effectiveness is the sole criterion for approval by the J0RP"3: 



1. Old a change occur? Was the change a positive one that was In some 
way related to the objectives? 

2. Was the effect consistent enough and observed often enough to be 

; statistically significant? , 

'3. Was the effect educat^lonally significant?. ». . In judging the 
,educat1onal significance of an Intervention's Impact, two factors 
must be considered: the size of the effect and»the Importance of the 
area In which It happened. There ought to be a reasonable balance 
between the two factors. The chance that a small gain' would be - 
considered educationally significant Is higher In a broad or 
educationally Important area than In a narrow or less Important area. 

4. Can the Intervention be Implemented In another location with a, 
reasonable expectation of comparable impafct? 

,1) 

Is the project setting unique? 

Is the project effect solely due to the unique characteristics 
' of the staff? 

What evidence Is there to suggest that the Intervention would 
work with different participants, in a different setting, and 
with a different staff? ' 

What components are essential? Can these be disseminated? 

5. How likely is it that, the observed effects resulted from the 
intervention? 

Can plausible alternative explanations be generated? 
Can the alternative explanations be rejected? 

6. Is the presented evidence believable and interpret aole? 

Are there any apparent inconsistencies in the data presented?^ 
Are enough data presented to satisfy the skeptical evaluator? 
Are the Inferences drawn from the data consistent with the 
evidence? 

Has evidence been presented that common errors have been 
avoided?^ 



Project staff who decide to submit their project for review by JDRP folJow. a 
specific format for submitting materials (see page 5). They can submit no 
more than 10 pages of explanation and documentation. The difficulty of 
selecting the most appropriate information and of demonstrating avoidance of 
typical evaluation pitfalls leads most project directors to hire outside 
evaluatorSi. 
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FOMAT m SUIMlTnM MTEXIALS . • * ' 
TO TNC JOINT OISSCMIMnON R£VICU P^BT 

mCRM ARCA: (t.^., TUIt III, rvtillfff, carttr tducttlOA, mrlronMnUl tdMCitlon, t^ucAtlon for tht 



I. tmmmon nru. wcatiom: 

Sp^fy tiM mirof tDt 1fit«nr«RtlQfi and ttw location for which tvldanct of tfftctlvmss Is haing 

II. ocmoro iv; * 

Indlcata «ho dtvtlopod tht Inttrvantlor oHjInally. tvon If this hamntd at a dlfftrtnt sitt than tht 
ont Hr which ovldonca of affoetlvantts Is bo1n9 prtsontad. 

iiK souAce MO isva or njNoiM: 

List all finding sourcts f^ tht Inttrvtntlon at tht location for which tvldtnct of tfftctlvtntss is 
prtstnttd and, for tach sourct, list tht aMunt of funds (stt Figurt 1 for an txaivlt). 

IV. TtAW OF INTrtVCirrtiJR OCmOFfOT: 

Indlcatt tht jftir or ysars during which tht Inttrvtntlon was originally dtvtloptd or tttttd. 

V. iMCF ocsoumoN or uiTcmirrioii: ' 

Irltfly dttcHbt tht Inttrvtntlon f^ which clalai of tffoctfvtntu art bting atdt. Tht dtscrlptlon 
should covtr at Ittn tht fallotHng ptints: 

What Is tht inttrvtntlon? 
Wiat art its ohjtetlvtt? 
What claffls tf t fftctl vtn M S art bting aadt? 
- Mitt Is tht ctnttxt In whia It optratts? 
Mho arrthtJnttndtd tistrs and btntfldaHts? 

Uhat art tht chtr»£ttHst1c3 of tht grotps on wMch tht Inttrvtntlon wu dtvtloptd and ttsttd? 

Ul4t art tht sniltnt ftaturts of tht Inttrvtntlonr 

Uhat art tht costs of adoption and Minttnanct of thtofnttrvtntlon? 

vx. mocNtc or crrscnvMss: 

OtscHbt tht tvldtnct of tffdctlvtntu for tht Inttrvtntlon. This stetlon should dtal with ttch of 
tht fdlloidng points, although not ntcttsaHly. in tht saot ordtr: 



Cvldtnct that tht quantltatlvt^Masarts art rtllahit and valid 
alMd. 



^i^^^lfi^^C^^^j^Wio celltcttd and analyxtd tht data* what assurtnctt art thtrt that 

^gitgl^^^^^^Miat Is tht tvldtnce thtt soutthlng h a pptnt d ? What art tht tfftcts clalMd 

gyldtnct o^«tat1tt1ci1 !t]<ffc<^<^,?^Jt7 t^ftcts: What Is tht avidtnct ^t tht tfftcts happtntd 
oftan mugrant with sufficitnt rtnatiiit/ to llltaly to happtn again undtr sisllar clmmtancts? 

evidtnct that tht tfftcts art tducation|P| tiinlnafyl Whit is tht tvldtnct that tht tfftcts art 
urot anouon, ptwtrrui tnougii, wr laporbni anough ^;obt tducatlontlly attnlngful, rtgardltss of 
thtir statlstlal sigtlflcttctf ^ 

Cvldtnct that thg tfftcts art attrlbutahlt to tht Inttrvyitltns Can alttmatlvt uolanatlons 
suea u prtctict trrtcts, naoirttion, stitctlon of suptnor trtatatnt groups,/«tc., bt niltd out? 

d1v«rw elmmtMca* to «1vt unrtnea that ttw affaetf cUimt my bt slalltr Wmh tiit 
predict or pnctlea Is uua tlMMtart for tht popHUtloM IntMdad. 
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The review process actually begins with the belief of a project officer that 
the project is. worthy 'of national dissenination* All submissions are reviewed 
in*depth by the originating federal program office. They are examined for 
factual accuracy, social fairness , and possible harm to users , as well as for 
evidence of effectiveness. The final review is conducted by a seven-member 
sub-panel of the JDRP convened by, the Executive Secretary. A vote is taken on 

o 

the submission immediately, after the review— a simple majority is required for 
a favorable decision. JDRP has been known to ask for resubmission where 
,evidi*nce is less than adequate for a decision. The minutes of aVl reviews are 
available to project staff and the general public. 

* o 

\ 

Projects that are'' approved by JDRP become eligible for, but are not 
guaranteed, dissemination funds distributed by the Department of Education to. 
selected projects. These projects become part ^ the National Diffusion 
Network, (NDN), a nationwide system iestablished to as^st schools, 
postsecondary institutions', ^nd ciphers in improving thel4:;education programs 
through the adoption of exemplary education projects approve^^y JDRP; such 
projects are known as Developer/Demonstrator (0/D) projects. W^th these 
funds, and with the assistance of State Facilitators (SFs)— offices located 
with each state and. funded by the Department of Education to help local 
schools and others learn about and adopt 0/D projects— the staff of validated 
projects can assist in the adoption or adaptatlor^^^^hetj^ programs or 
practices at other lites. D/D projects ve described in tl^ NDN catalog. 
Educational Programs that Work , which is distributed nationally. 
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^ Identification. Validation, Dissemination (IVD) 

• The IVO process was Initiated Jointly by the National Advisory Council for 
Supplementary Centers and Services, State ESEA Title III Coordinators and ESEA 
Title III personnel In the U.S. Office of Education, and the National 
Asfoclatlon of State Advisory Council Chairmen, with Input ard assistance from 
the^^State Departments of Education. Although IVD was Initially designed for 
the validation of Title III (now Title IV) programs, the developer; hoped that 

« the proc.ess would be used for validating other programs as well. In fact, ' 
most states that use the IVD process use Title IV funds to validate programs 
funded under that Title. - 

' . The IVD process represented for Its developers a quality control effort which 
would serve a dual function: 

First, .It would provide a systematic mechanism by which the 
educational value and effectiveness of emerging programs, practices, 
and products could be reviewed and assessed as to* their. sticcess; 
I.e., the purpo;ses and outcomes of developmental 'projects would be 
"proven to work." Second, the validation process would enable the 
crieation of a hank of proven education^ practices. Once programs 
and products have been validated as effective and exportable, they 
could be entered into this bank of successful programs. The 
validation program, therefore, addressed the need to identify and to 
certify programs and practices ,that could facilitate constructive 
educational changes in our nation's public and non-profit private 
schools, and provided guidance riot only in establishing the 
educational worth of a project at its original site but also in its 
successful replication in other schooT systems.'. 

The IVD process is guided by a handbook called Sharing Educational Success; A 
Handbook for Validation of Educational Practices . The original handbook made 
very explicit the procedures to be used by the-state, by the validation team, 
and by projfjct staff. It offered three very general criteria, but the forms 
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In. the handbook made very explicit the requirements for meeting these 
criteria. Projects were. given no guidance In Sharing Educational Success 
regarding evaluation haiards or kinds of data that represent proof of 
achievement gains. 

In 1979, a revised edition of Sharing Educational Success was completed by the 
U.S. Office of Education in cooperation with the Assoclatloh of State^Advlsory 
Counc'lls and the State Departments of Education. Three major considerations 
led to the need for a revised edition: (1) the states felt the need to 
develop a cadre of Individuals at the state level who would be well trained 
and who would ensure quality control of the validation standards nationally; 
(2) there was a need for state autonon<y In the validation process>«autcnomy 

o 

which would allow the states to modify, within federal guidelines, processes 
and procedures for project val1d.at1on; and (3) there was e need to reduce the 

g 

cost Of the IVO process for both the states and the federal government. 

In the IVO process, each state validates projects that It believes ar^ 
e'><ei'nplary, based on the following criteria: 

1. Effectiveness/Success: Project objectives Identified for validation 

are supported by convincing evidence showing statistically and 

educationally significant outcomes* 

The documented effectiveness or success of a program or practice 
is of parvnouiit importance for , validation. A program or 
practice can b,e *^proven to work" in numerous ways including: 
. , (1) by demonstrating with convincing evidence that the program 
will bring aboitt desired change or improvement over .the existing 
practices, (2)iby demonstrating a more efficient or cost- 
effective program or practice through Improved management, 
' resource utilization, etc., or (3) by demonstrating with 

convincing evidence that a desired objective may be accomplished 
without detrirrA^nt to the existing program. 9 
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2. Export ability: Information Is provided to demonstrate project or 



practice Is capable of being diffused to other school districts and 
can be adopted or adapted by other school districts with similar 
needs and environments; 



For the project as a whole (or for each applicable component) 
Information required includes evidence of educational 
significance, a description of the minimum level of adoption or 
replication which would produce similar results, and information 
about the target population; staffing and training requlremients; 
materials, equipment, and facilities; replication costs; and 
special problems.'" 



Sharing Educational Success describes six steps to be taken In the validation 
process. These steps can be modified by Individual states to meet their own 
needs, and time- and money-saving options are 'suggested. 




U LEA completes and submits application for validation to the state 
agency or office responsible for coordinating validation activities 
at the state level. 

c. Preliminary review by the state agency for validation followed by: 

a. approval for validation team review; or 

b. return jto_the LEA for revi s Ion accord Ing to suggestions ; 

c. disapproval for further validation.- 

3. Selection of the validation team: 

a. the team leader to be selected from out-of-state from the 
list of USOE -trained team leaders within the region, 

b. two team members selected from within the state from the 
list of State-trained numbers. 

,4. Review of application by Individual team members. (This may be done 
as individuals 1n isolation from the other team members or the team 
may be convened to review an Individual application or serve as a 
panel to review a group of applications): 

a. the team leader makes reconmendatlons to the state agency 
responsible for validation that the Identified revisions 
are to be completed before the on-site team visit is 
conducted. : 

b. the teamJ[eader1nfo^rms^the state agency responsible for 

vaTldatlorTtMt the on-sTte visit be conducted according to 

the existing application. 
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Conduct of the on-site visit. The most frequent procedure 1s to send 
all team in«nbers on-site. Some states have designated an Individual 
team member to conduct the on-site visit as a cost savings measure. 
The individual conducting the on-site visit serves the role of 
collecting and clarifying any Incomplete or missing Information 
Identified by the review of the application by the whole team. 

6. The primary 'decision of the validation team Is either approval or 
disapproval under the IVO standards. 

If approved the team might also make the following recommendations: 

1. Submission to JDRP 

2. State Dissemination 

3. A special comoonent or product be recognized as worthy of 
distribution. 1' 



1 

An abbreviated form of the application requirements for IVO validation Is on 

pages 13 and 14. The requirements set forth In the original edition of 
Sharing Educational Success , again In abbreviated form, can be found In the 
Appendix. 



The IVD process requires no commitment from the federal government beyond 

assistance In training validation teams. After a program or practice Is 

validated, the developers of Sharing Educational Success recommend that the 

state take one or more steps: 

Successful completion cf the federal-state IVO process would be tied 
to a "pay-off to the project and to the state. Such rewards may 
Include, but not be limited to, entry Into a state-operated 
diffusion network, access to funds for project diffusion, 
preliminary screening for national JORP validation, or to otherwise 
. serve as a. vehicle for educational Improvement within the state. 



The Department of Education may furnish technical assistance for state-level 
diffusion, but only If ^ state-validated project Is also approved by JORP, 





State Variations . The IVD process has been somewhat controversial since Its 
Introduction. Some states still use this process, some states have modified 
or adapted the process to meet their own needs, and a few have no validation 
procedures at all. 



An August 1977 survey of certification methods applied to innovative public 

school programs conducted by Ray E. Foster of the Educational Innovations 

14 

Section of the Florida Department of Education found that, of 50 states 
and territories responding to the survey, 45 responded positively to the 

following question: 

Does your state education agency use a formal method to review and 
.certify project developed programs {e.g., those developed under ESEA 
IV-C Innovation) before promoting their widespread use?"' 

A total of 33 respondents (73X) reported using the IVO process, although only 
5 of these used that process alone; 22 used the IVO process and the JORP 
process; and 6 used those two processes In conjunction with a state-developed 

process. Six states Indicated discontinued use of IVD, and five states 
that were using the IVO process at the time of the' survey Indicated that they 
were In the process of developing their own certification methods. 

A conclusion of the team conducting the survey was that "It appears that the 
current trend Is toward state-developed certification methods and away from 

IVD participation."^^ 

A more recent survey, conducted In 1980 and 1981 by the Research and 
"Developrnent~Excliange, shows that the state of affalrs has-not changed much 

11 J5 : 



since the 1977 survey. Of the fifty states surveyed by the ROx, all 
responded. Forty-five reported that they have some form of established 
-procedure for validating promising educational practices. Alabaina, Hawaii, 



Louisiana, Mississippi, and Nevada reported that they do not have a 
procedure. Approximately two-thirds (29) of the states that have validation 
programs employ either the IVO procedure (23 states) or some modification of 



the IVO procedure (6 states). Typical modifications of IVO involve the use of 
in«istate site reviewers to reduce costs and/or the use of state review panels 
In conjunction with the site visits. Roughly a third of the states (N"16) 
reported that they use their own state-developed validation processes. The 
state-^eveloped-processes differ from IVO with respect to the specific 
crittria and operational procedures used to verify the effectiveness and 
transportabjlllty of the promising practices. Two of the states with their own 
processes (New Hampshire and Pennsylvania) viewed them as modifications of the 
JOftP process. 

Sharing Educational Success mentions two benefits which accrue to states which 
develop their own validation process: 

A few states have developed a state-level IVO process which draws upon 
many of the significant elements of both IVO and OORP, but represents 
■more than a composite of tht»se two processes. In general, the state 
IVO process attempts to develop a state-administer^ procedure for the 
quality controT of programs, practices and products that are consonant 
with and responsive to the needs of an individual state. Further, 
because the validation process involves the participation of 
Individuals and projects th^t come from a more limited geographical 
area, the major benefit expected to accrue Isjthat the validation ' 
process promises to be more cost-'»ffective. 
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b* Actual changt or rasults* Utillzt charu« graphs, statistical swMrlts khtrt approprlatt 
c. $1gn1f(canca of rtsu1ts-«t1thtr statistical or othtrvlst. If othtr than lUtlstlcal pt^dt 
rttlontit for tvtluatltn of signlflcanct* 

Ftr jMtrtll praJtct-rttulU: 

a* Estfflttt of aduatlonal or prtctlal signlfleanct of findings 
b. Irltf intarprtatlon of rtsulu 

2« Shot thtt tht rtsulu want, systanatlally and coapttintly analyztd. 

3* Aaptrt unantlclpactd outeoats of tajor Itportanct and signlfleanca. 
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7, 
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\ 
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Is SMCh 



4. >trt»HiU. liylMint. raelHtlM 
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lovot of-rtplicatton of your profras ttet «oo1d product 
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Sharing Business Success (SBS) 

Business practices which successfully go through the procedure outlined In A 
Handbook for identification. Validation, and" Dissemination of School Business 
Practices receive, validation approval from both the State Education Agency and 
the Association of School Business Officials. The Process is similar to the ^. 
Identification, Validation, Oisseminatioin (IVO) process in operation but is ^ 
similar to the JDRP process in its rigor. 



AppTl^^s for validation are required to submit a preliminary application 
which 1$ rev1e»i»ed by a screenNog committee. Applicants supply the following 
information: (1) a statement of major outconws in objective form; (2) a 

r 

description in sequence of' the changes, interventions, or activities whch 
caused the accomplishments or outcomes; (3) a description of how the outcomes 

4 

0 

.were evaluated and evidence that achievements were significant, the result of 
the new practice, and persistent over time; (4) i desci^iption of, materials 
and/or equipment used by the practice; (5) a statement of all costs; (6) a 
~dtscription of savings in time and/or money; (7) a description and number of 
personnel directly affected or Involved, in the practice; and (8) a statement 
about other relevant information or special conditions bearing on the success 



of the practice. 
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Those local education agencies which are invited to make full application must 
supply extensive information about the practice— its effectiveness, 
efficiency, cost, and expoftability— and roust undergo at least a one and a 
half day site visit. The criteria for success are those used by the Joint 

f 

Dissemination Review Panel (JORP). 
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An abbreviated fom of the application for validation can be found on pages 
17-19.^^ 

Project Informal-Ion Packages (PIPs) 

Another major validation effort, undertaken by RMC Research Corporation for 
the USOE Office of Planning, Budgeting, and Evaluation In the early seventies, 
was a project called "The Development of Project Informiitlon Packages (PIPs) 
for Effective Approaches In Compensatory Education." The concern that the 
projects selected for replication under this project be "exemplary In 
producing significant cognitive achievement benefits" led to a major effort to 
develop and describe a validation process more stringent than that used by the 

22 

JORP. *■ Approxirately 2,000 projects In compensatory education were reviewed 
by RMC In a nationwide search. Of these, 103 were examined In depth; only 6 
met all of the criteria. 

An outgrowth of the work performed by RMC for the PIP program was the 
publication by USOE of two volumes In a projected series of handbooks designed 
^ to provide Insight Into validation procedures. The two volumes which have 
been published are A Procedural 6u1de for Validating Achievement Gains In 

23 

Educational Projects and A Practical Guide to Measuring Project Impact on 
Student Achievement ,^^ which discussess five evaluation models and the 
procedural Implications of each. 

Selection of projects for the PIP program Involved three levels of criteria.- 
In the first screening, designed to eliminate programs which were not 
compensatory, four prerequisites had to be met: , 
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1. Provides instruction in reading and/or> math. 

• 2. Serves children in grades K-12, 

3. Serves educationally (Usadvantaged children. * * 

• 4. Has achievement test oJba for more than one "instance." ' * 

The second level of screening centerted on the folloing considerations:. 

1. Availab'ility . . . ' 

Accessibility (can be visited for validation; personnel are 
cooperative; procedures, results, and costs are documented), . 

\ . 

Acceptability (operational .In public schools; hot primarily a,'- 
~ single commercial product) 

2. Cost - , / • 

Cost of equipment plus, special personnel (ppr pupil) 

Cost- of initial Investment (per pupilj) - - 

3. Se^>rtca^11ity . ' • . . 

-^11 major components can be clearly duplicated ^ 

4. &fectiveness . ^ 

Achievement test "data show conslstetftly that actual post^ 
treaianent performance exceeds the no-treatment expectation by an 
amount. which! is statistically significant and equal to at^east 
one-third standard deviation. With respect to thejiational norm. 
A gain of one-third standard ilevlatlon. with respect to the 
national norm was chosen as the criterion to be used in the 
national .packaging effort for determining exemplary status. 25 

The final stage of screening for the PIP project required descriptions of the 
experimental and comp^arison groups, the evaluation model employed, discussion 
of possible confounding influences, and representation of evidence of 
' statistical significance. Projects which were still under considerajbion after 
the final screening were then taken through the 22 steps in the "decision 
tree" for validating statistical significance which is described in A 
Procedural Guide for Validating Achievement Gains in Educational Projfejts . The 



■ ^ 
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three screening workshops, the flow chart for the decision tree and a 

trief description of the 22 steps^^ are on^psges 22-27. ^ 



National Education Practices File 

iNIE^s Educational Resources Information Center (ERIC) has also responded to 
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the growing concern about Identification of worthwhile educational programs 
^ practices. The ERIC system was originally designed to make readily 

« 

Available fugitive research, conference documents, and fe<lera1 reports. In 



the early seventies ft became evident that many of the documents In the system 
\ ■ • ' 

eni^islred practice rather than research, and the focus for the monthly 

jcHimaK . RI£i changed from Research In Education to Resources In Education . 

'^'Thls was not^odgh, however, and thei}ressure to develop a file of 

educational practices grew. A feasibility study for such a file by . . 



Bibllographfc 
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Retrieval Services, Inc. (BRS) led to the design and development 
Izatlon of the National Education Practices File. One of the 



major problem^' for 8RS In developing the file was. the question of how to 

.validate the practices. In fact, that proved Impossible— some practices In 

\ 

the National Education Practices File have evaluation data, others do not. 
Decisions about the relative merits of the practices are left to t\e users of 
the file. 



-<r ' 

In January 1979, NIE selected 12 pilot sites for the National Education 
Practices File: 



Alaska State of Alaska, Department of 

Education, Juneau 

California San Mateo Educational Resources Center 

(SMERC), Redwood City 
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Illinois State Board of Education, Illinois 

Resource and Olssemlnatldn Network, 
Springfield , 

Maine Hid Cost Teacher Center, Camden 

Michigan Wayne County Intennedlate School 

District, Wayne 

Minnesota Exchange at the Teacher Center, 

Minneapolis Public Schools/University 
of Minnesota, Minneapolis 
Missouri University of Missouri-Kansas City, 

University Libraries 
Ohio ERIC Clearinghouse for Science, 

Mathematics, aitd Envlpnmental 
Education, Ohio State University, 
Columbus 

{ \ - Oregon ^ Nort^hwest Regional Educational 

Laboratory, Portland 
Texas Southwest Educational Development 

V- Laboratory, Austin 

Texas Texas Education Agency, Austin 

Washington, O.C. National Institute of Education, 

Educatlpn'jil Research Library 

The pilot test began In May 1979,. with a collection consisting of 

1. All programs d1ssem1naC(sd by NDN (and therefore validated by JDRP). 

2. All Title IV-C programs for which SEAs submitted descriptions (some 
of which had been validated by IVD, some by stat^ processes, and some 
unvalldated). 

— J. Prograjns In mathematics, science, and environmental education that ° 
were developed by R&O labs and centers and by practitioners (with a 
variety of evaluation data, some empirical, some subjective). 

In f^all 1980, the National Institute of Education decided not to fund a full' 
1mp1@;;entat1cn cf the Natlonl Education Practices rile. Tne Education Service 
. Group, Inc., of Scotia, New York, Is Involved In an effort to continue the 
work begun during the funding period. The "School Practices Information File" 
Is offered as part of ESG's School Practices Information Service. 29 



ERJC QQ 



other Significant Efforts 

A search for ^enqjlary projects In the area of reading was conducted by 
American Institutes for Research (AIR) In the early seventies for the Right to 
Read Office. The search Involved reviewing past research studies, 
computer-scored abstracts, add library materials, as well as obtaining 
-nominations from experts In the field and from staf^f members of educational, 
-professional « and goverrenent organizations* -Information about each program 
was gathered by means of. a study questionnaire, the Program Information Form 
(PIF)i and from mdrH detailed program documents. 

Over 1,500 program candidates Identified through literature searches and 
\ nominations were sent the PIF; 728 completed forms were returned. The 
criteria for screening the reading programs Included 

1. Location within the United States. 

V 

2. Operation for at least one year and the expectation of two additional 
years of operation. 

3. Availability of program evaluation evidence reported since 1968. 

4. A focus on reliably measured reading achievement. 

5. An adequate assessment design artd statistical treatment comparing 
reading achievement gains for program participants with ga1i?s for a 
credible non-participant group. 

6. Adequate size. 

7. Potential for replication. 

AIR reported that, ult1mat€».ly, the quality of local program evaluation was the 
major discriminator between effective and non-effective programs in this 
massive search for effective reading programs. 30 



Twenty-seven of the 728 programs were recornmended by AIR as having exhibited 
'positive rea\(1ng galr^s on the part of students over a certain number of 
years. Of these twenty-seven, twelve were selected by Right to Read based on 
the criteria of the Joint Dissemination Review Panel; these were to be 
packaged and disseminated Individually. 
• * 
Right to Read felt that many of the remaining programs for which forms had 
. been returned. also, showed potential for-lmprovlng reading ability. "T^ken 
together, these p!*ograms represented a wide range of approaches to teaching 
reading and provided a wealth of Ideas that m1ghtl)e adapted 'to fit local 
objectives and needs. These programs a^e described In a separately published 

31 

catalog Effective Reading Programs: Summaries of 222 Selected Programs . - 

/ 

t 

The Office of Dissemination and Resources of the National Institute of 
Education also launched a search for promising programs and practices. The 
Catalog of Promising Educational Programs and Practices, compiled by Automated 
Services, Inc., was. a collection of abstracts describing selected programs 

t 

operating In the nation's schools during the 1972-73 school year. 
Approximately 3,000 programs were Identified and surveyed by malgl-. Of the 
1,000 programs' which responded to the survey, 1^7 were chosen for Inclusion In 

32 

the Catalog .^ 

The Catalog offers another example of an early attsnpt to estauMSh validation 
criterlja. "Verification" rather than "validation" was the word used In this 
projfxt: 
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1. The program has specif k, identifiable educational objectives. 

2. The program has been in existence for at least one yean. The local 
school should be planning to continue it. 

3. The program has been evaluated and has been judged to be at least 
"promising" or "verified." The term "promising" is used cover- 
those programs showing potential for producing more effective 
results, but for which convincing evaluative data are lacking. At 

. the other extreme c "verified" is used for programs supported by. hard 
' evaluative_data_on effectiveness. 

.4. .The<^loca1 district sponsoring the programs should be able and willing 
to respond to any Inquiries on the progrfm/practice that national 
disSjMilnati on .may generate. - 

> • « ■ , V 

5. The program has the potential for successful replication In settings 
other than those for which it was originally designed. 

6. The program is not dependent for success on any special feature, such 
as a charismatic director, expensive equipment not usually found in a 

0 local school, or exorbitant costs. 



MAJOR ISSUES IN VALIDATION 



States which are developing or refining validation procedures must grapple 
with many issues. A description of some of the major issues follows. 

Replication vs. Adaptation s 

Is it likely that an exemplary program or practice which is chosen for 
adoption by another uchool district will be replicated, or will it be adapted 
to meet the needs of the students and staff in that district? Obviously, 
there are proponents on both sides of this argument. Two examples of programs 



which their developers feel have led to replications all over the country are 
the National Diffusion Network (NDN), consist1(ig of JDRP approved educational 
programs and practices, and the Experience Based Career Education Program 

(/bce). 



\ 

\ 
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The HON has been discussed earlier In thfs papeV. Evaluation of that network 
ttas shown that replication has occurred: 

the approach appears highly effective In creating lEA awareness. 
Interest, and subsequent Implementation of Innovations being 
diffused. Factors accounting for successful adoptions were: (1) 
thorough, persistent, and wellHllfferentlatecl awareness activities; 
(2) early Involvement of administrative and Instructional decision 
makers and emphasis oh local commitment; (3) extensive use of 
Ih-person -tactics at sll stages of the adoption-process. Including 
visits; (4) Provision of comprehensive and well -developed 
materials to support adoptions; (5) personal dynamics of the 
Oevelcper. and management skills of the Facilitator; (6) emphasis on 
pract1,t1orter change,,,phase-1n of Implementation, and low reliance on 
expensive resources; (7) support of' LEA visits tovdembnsti^ation 
sites. Adoption patterns appeared reasonably uniform by innovation 
type and 4ieogriiph1c area, but disproportionate in terms of school 
level andldlstrict urbanism. Much of the overall NON success is 
attributed to effort and enthusiasm of participating change agents.^^ 



With support from the Nationl Institute of Educatjon, four different EBCE 
O programs using the community as the classroom were developed between 1970 and 
1975 by educational research and development groups (the Appalachia 
Educational Laboratory, the Far West Laboratory, the Northwest Regional 
Educational Laboratory,, and Research for Better Schools). In 1975, following 
/a laudatory external review by practitioners, researchers, and experts in 
educational change of the programs' in the four develop.>ent sites 
(Philadelphia, Charleston, Portland, and Oakland), EBCE was> approved by the 
Joint Oissemination Review Panel. 



One component of the dissemination strategy was selection of several pilot 
sites for each of the four EBCE models. These sites received very extensive 
technical, assistance from Developer/Demonstrators (50 to 80 person days for 
each of two years). Alf operational costs, however, were borne by the school 



rsystews In the pilot sites. These "high fidelity" sites. were intended as 
places for the program developers to learn h^w to implement their programs 
eUe^re, to see if the results would be as positive as they were In the 
original four development sites, and as training and observation sl^es for 
others'.' All people who were interested in the progress of the network could 
learn more about it through n^sletters and conferences. Expansion of the 
network include d $100.000 plannin g awards to four states^ free technical 
assistance to thirteen other states, and .dissemination of the program by the 

j U.S. Bureau of Aciult and Occupational Education. In 1976 and 1977 EBCE's were 

1 ■ ■ ~ ' Vs' ~ " • ' ' 

.jjinplemented in 44 states and territories.- 



One of the most important recent documents addressing the question of 
replication vs. adaptation is Federal Programs Supporting gducational'ttfange. 
the 8-volume. report of a four-year study conducted by the Rand Corporation. 
Only a sampling Af their findings is possible here. 



-One important finding, or perhaps we should call it a confirmation, Is that 
LEA staff tend to seek solutions to their problems from local sources: 



Our evidence suggests that the "search for alternatives" traditionally 
assumed to be characterise of the problem-solving approach to innovation 
did not occur. In developing projects, LEAs used Information or 
treatments that wer$ already kno^n to local district personnel. This may 
mean. that LEA staff Intuitively feel that the success and suitability of 
an innovatioh. depend primarily on local conditions, a view that out* 
evidence supports. . Thus, local administrators are likely to be skeptical 
about the reported "success" of educational methods in .ather districts and 
tend to rely on the advice of local professionals who have a thorough 
knowledge of particular local conditions. '^'^ 



The Rand research group observed three types of interactions— "defined by the 
extent to which the project was adopted to the institution or vice versa"— 

that characterized implementation, processes: 

<^ . • ' 
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1. Wonlwplewentatlon occurred when the project neither-altered its 
setting nor was adapted to It. Some projects simply broke down 

' during Implement at ion i particularly If they were very comprehensive ' 
or "overly planned* and prescribed; others were Ignored or received 
scant attention from: users, particularly if they had objectives that 
were trivial or peripheral to classroem concerns. 

2. Cooptation occurred where the staff adapted, the project, usually , 
, emasculating it, to meet thtir own needs, without any corresponding 

change in traditional institutional behavior or practices. Such 
projects could experience a deceptively smooth implementation. 

3«- Mut6al-adaptat1on" Occurred"When- both project and setting were 

changed, nutual adaptation could 'InvoTve a variety of adjustments to 
the project Itself— for example, reduction or modification of 
1 deal 1 st 1c vpro ject go al s amendment . or .^s impTI f 1 cat i on of project 
trettmentj, downward revision of ambitldu's expectations for behavioral 
change in the staff or of overly optimistic effects of the project on 
students, and so on.. Concomitant with these modifications in projisct 
design or objectives, new behaviors were required by project staff, 
as well as hew attitudes necessary for integrating project strategies 
into classroom practices. Mutual adaptation seldom meant smooth or 
trouble-free Implementation. Indeed, from the perspective of an 

~ — outside observer, the first year-or sa of project operations^ might 
often be seen as chaotic, as staff tried hard to make the project 
work for them.37 . 

Not surprisingly, a project's outcome depended on the process 
characterizing its implementation. Projects that either broke down or* 
ware applied In a pro forma way (i.e., were essentially nonimplemented) 
caused little change* in teachers and little Improvement in student 
performance; they wore generally perceived as achieving a low percentage 
of their g6als. Coopted projects not Infrequently achieved an average or 
afetove average percentage of their goals, depending oh 'their complexity and 
' ambit iousness, but they generally did not significantly alter the 
tcachei^s' behavior.] Projects whose. Implementation is best characterized . 
as mutual adaptation were not Invariably successful, particularly when 
their ambition surpassed their capacity. Yet, they had a better chance of 
being effectively implemented. Moreover, mutual adaptation was the only 
process leading to. teacher change; in other words, teachers changed as 
they <and only as they) worked to. modify the project's design to suit 
their particular school or classroom. 

Ut can understand why mutual adaptation char&cterizes effective projects 
by looking more closely at implementation In the classroom. The task of 
teaching essentially consists of a one-to-one relationship between teacher 
and student, it thus necessarily depends heavily on Idiosyncratic teicher 
and situational characteristics, and consequently, the. same projec.t will 
be implemented somi^what differently in each classroc;?! and in each school. 
If Instead a project is' applied uniformly or rigidly. It is unlikely that 
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the new techniques wHl significantly or positively alter the teacher- 
student relationship. The development of new teaching behaviors within 
each classroom is a pragmatic, learn-by-doing ^process consisting of a 
step-by~step fine tuning of project design. Therefore, the process that 
fosters effective implementation and teacher change is one that promotes 
each teacher's ability, capacity, and mo*>ivat1on to accomplish this unique 
tuning. 38 



Fullan and Pomfret (1977) offered their perspective on the question of 

replication vs. adaptation after analyzing fifteen studies on the problem of 

defining and measuring Implementation: 

In General, current central policies (at both the national and school 
district level) at best promote adoption.* These poHcies do not have a 

• strong Influence on implementation because they fall to address those 
\ fjictors most critical for implementation. In order to support more 

effective implementation some-major Interrelated steps are necessary- 
First, Instead of promoting specific innovations, central policy makers 
should be emphasizing. broad-based programs and providing corresponding 
support forJocal development of specific forms-of implementation, thereby 

* facilitating clarity and explicitness of programs on the part of users. 

. Impalement at ion plans could be requested in advance from local authorities, 
and could be monitored with the expectation that specific innovative 
projects would be Implemented and assessed. 

Second, local experimentation should be encouraged during Implementation 
to develop variants of innovation in which specific goals and means are 
seen as consequences of exploration, negotiation, and "development in use. 

Third, evaluation of innovative projects at least during initial 
implementation should be directed toward facilitating implementation and 
local system capabilities through data feedback and other forms of. 
support, rather than toward judging success or failure. 

Fourth, and closely related to the previous points, the incentive system 
for implementation Muld have to be drastically altered at all levels. If 
there is one findlAg that stands out in our review, it is that effective 
implementation of social innovations requires time, personal interaction 
and contacts, in-service training, and other forms of people-based 
support. Research has shown time and again that there is no substitute 
for the primacy of personal contact among implemonters, and between 
implementers and planners/consultants-, if the difficult process of • 
unlearning old roles and learning new ones is to occur. Equally clear is 
the absence of such opportunities on a regular basis during the -planning 
and Implementation of most innovations.. All of this means that new 
approaches to educational change should Include longer time perspectives, 
more small-scale intensive projects, more resources, time, and 
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mechanisms for contact among wou1d«be Implenenters at both the Initiation 
or adoption stages, and especially during Implementation. Providing these 
resources may not be politically and financially feasible In many 
situations, but there Is no question that effective Implementation will 
not occur without them.^^ 



Evaluation Experience. of Project Staff 

All of the validation procedures that have been developed have been Intended 
as guides for interpreting data that have resulted from both experimental and 



quasi-experimental research designs. tShey are not intended as guides for 
conducting evaluations, although an understanding of the process, the criteria 
used in validation, and the potential evaluation pitfalls Is critical for the 
project director who intends to try for validation and for the State Education 

« 

Agency or- Intermediate Service Agency personnel who will offer technical 
assistance to the project director^ as he or she l)repares~to' submit the project 
for validation. 

V ' ' ' 

A major problem for states as they use IVD or develop their own validation 

procedures is t::e inability of most LEA personnel to deal adequately vfith the 

-evaluation question as they 'plan and execute the research design. The 

National Association of State. Advisory Councils (NASACf study of the IVD - 

process addresses this problem: 

Most personnel directly invovled with a local education agepcy seeking 
validation of its project have virtually rra prior experience or expertise 
in evaluation. This consensus opinion is especially portrayed by project 
. evaluation efforts associated with the design and conduct, analysis and 
reporting of project outcomes in a form likely to be acceptable to the 
validation team, ftoreover, very few LEAs have the internal capabilities 
to assure an evaluation of a project prior to }ts submission for 
validation. 

* ' 

The majority of projects are therefore required to seek external support 
for evaluation assistance. In general, the LEAs engage evaluators with - 



ERIC 



36 41 



varying degrees of i expertise 'in program evaluation and, ideally, with 
.^experience in the requirements of an IVO assessment, per se. The major 
implication of this decision is that the notion and purpose of evaluation 
takes on a very narrow objective in itself: assuring a successful 
validation o"nly." ^Most project directors .interviewed by the study team 
reaffirmed that, to successfully enter and "pass" IVD (and the JDRP 
review, if sought), the contracting of an experienced evaluator is. . . a 
"no-choice" segment of the decision process.*" 



Many projects which come before JORP for validation- are rejected because of a 

jglAw^n.,the,reseaKCh, design--or_evaluation-pIan , acc or d iiig-^ Seymo;;tr Rubakr - 

41 

past Executive Secretary of the JDRP. Or. Rubak suggests that project 
staff become aware in the project planning stages of potential pitfalls and 

guard carefully against them. Hiring an evaluation specialist after the fact 
cannot eliminate these flaws. The JDRP Ideabook lists 14 of the most common 
evaluation hazards and describes how to avoid them. The flaws are listed 
below: . » 

1. Claiming much, providing evidence of little. 

Where evidence matches the claims, a favorable decision is far 
more likely than where evidence falls far short of- goals, 
objectives, and claims. 

2. Selecting measures not logically related to the intervention. 

3. The use of grade-equivalent scores. 

Grade-equivalent scores provide an insensitive, and, in some 
instances, a systematically distorted, assessment of cognitive 
growth. 

4. The use of a single set of test scores for both selecting and 
pretesting participants. 

. 5. The use of comparisons with inappropriate test dates for obtaining 
information. 

In norm-referenced evaluations, tests should be administered at 
nearly the same time as the test publisher tested the norm group 

■6. The use of inappropriate levels of tests. 



7. Missing data. 

8r"The use of noncomparable treatment and control groups. 

9; The use of Inappropriate statistical adjustments with non-equivalent 
control^ groupsTk 

Making betiifeen-group comparisons using either 7raw" gain scores 
or "residual" gain scores should be scrupulously avoided. 

.10. Constructing a matched control group after the treatment group has 
been selected. 

n. Th4 careless collection of data. 

12. the use! of different Instruments for pretesting and posttestlng. 

13. The use of Inappropriate formulas to generate.^ no-treatment . 
exptctatlcni. - . 

. Many projects use an unrealistic theoretical model or formula to 
calculate "expected" post test scores from IQ or other pretest 

s cores. If s tudents do better than the calculated expectation. 

the: project fs considered a succes. 

H. Mistaken attribution of causality. 

The- plausibility of alternative explanations should be carefully 
~ — examined before evaluation results are attributed to project 

Impact, as evaluation hazards are often the cause of apparent 
gains or losses.^2 

One example of an attempt to educate project_staff In the early stages of 
project planning can be found In the Right to Read Validation Procedural 
Guide. (Right to Read encourages all Interested LEAs to have their reading 
programs validated.) In addition to a description of validation procedures, 
the Guide Includes a description of evaluation hazards, a four-step 
experimental design, an explanation of standardized tests, a table of 
representative learning outcomes and possible methods of evaluation, a sample 
assessment planning chart, and an evaluation checklist. 



state Responsibilities Before Validation 

Is It the responsibility of the state to educate project directors concerning 
the pot'entlav hazards In^'prbject design and evaluation? The executive summary 
■of the NASAC-IVO project describes what seems to be the process In the five 
IVO states which were studied (Colorado,. Florida, New Jersey, Minnesota, 
Washington): 

Th* cWlQtirstTvewrel)f S£A^ersTnl!eT,^pirricuTarTy^h^se as witfr 
Title IV7C offices, can be extensive or minimal. The degree of SEA 
Involvement In the decision process appears to be highly. dependent upon 
the level of expertise of the project director. . . . [S]ubstant1al 
cbn'siiltfttlVe support Is directed by the SEA to those projects which do not 
appear to be sufficiently aware of the key elements— and pitfall s~of tne 
validation process. 

Key projects and programs known by SEA officials as having a high 
probability of successful development and validation are often • 

*pre-$elected»-by-the-SEA-:as-11ke1y-cand1dates-for-IV0i-or even-JDRP — 

.review. The probable educational significance and exportablllty of the 
project to other school districts are key a priori benchmarks In the 
selection process. These projects, once selected, are then subjected to 
,.-carefu.l "care and feeding" throughout the developmental stages leading to 
. the decision to seek validation, and Indeed through the validation process 
Itself .43 



"Exemplary* vs "Promising* 

A critical Issue for c'lates developing a validation process Is the degree of 

"rigor" which should be applied In validating* programs. In "Exemplary 

Practice: A Report on Recent Searches," Turnbull addresses this question: 

Much controversy around criteria centers on the word "rigor," which tends 
to mean a requirement for hard evaluation data. Advocates of rigor 
maintain that without requiring date) there is no way of being sure of a 
program's effectiveness. The most stringent searches for exemplary 
programs have rejected hundreds upon hundreds of programs because they ' 
lacked satisfactory evaluation designs or results. 

Some critics contend that it is wasteful to locate these programs and not 
publicize them, since they might contain valuable ideas. Moreover, what 
will be the local repercussions when a program is considered and 
rejected? Other critics say that projects should not be penalized for 



39 44 



local evaluators* frequent inability to meet strict standards. A' similar 
point is that local decisionnnakers do not want to apply these standards 
in screening projects that they might adopt.*^ 

The cootroyersy concerning the question of whether programs with no hard , 
evidence of impact should be selected and disseminated was also addressed 
during the July 1974 NIE Conference on Increasing the Use of Promising 
Practices Information by'Local Education Agencies. Participants at the 



conference— local educators, SEA personnel, and information services 
specialists— arrived at the following understandings of this controversy: 

1. LEA'S do have, the iresponsibllity for finally judging a program's 
*(brth.. 

2. Promising practices information is'Useful to LEA's when considering a 
Wide range of ideas, alternative approaches and new programs focused 

i n a s ingl e probl em area, as well as when seeking a single, J 

'fexemplarF' model. 

3. Funds are not presently available to validate more than a handful of 
profflising practices.- Thus, liard data on outcomes will not exist in 
the-near future for most promising practices, "for reasons entirely 
unrelated to the valuator "promise. "^5 



Many states have tried tc a.ddress these problems by developing their own files 

of exemplary programs and practices. A 1978 survey by BRS revealed that 97X 

of the 37 states and territories responding to the survey were building (83X) 

or were intending to build (14X) an educational resource base: 

The focus of content is on projects/practices somehow associated with a^ 
"quality" measure, be It a defined orocess (validation) or a more 
subjective selection ("exemplary").'" 



There is inherent in these activities a potential coordination problem if 
states do not take some precautions during the planning stages to set up 
knowledge bases that will be compatible with the knowledge bases of other 
states and, perhaps, with, the National Education Practices File. Gregory 
Benson expresses this concern in his report of the BRS survey: 
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A move toward effective educational practice "labeling laws" would go a 
long way toward eliminating many of the current "linker" activities 
required now because we do not know or report^ sufficient content, context, 
l^d/pr' contextual quality Indicators regarding educational practice. We 
shiMid move toward resolution of these Issues together^ and the one 
e1eii«nt common to all-educational change efforts Is the knowledge base, 
individual lyV wte are reftll zing this fact and attempting to do something 
about It. . . However, If we are to develop a national capability 
reflective of our Individual needs, there must be a mechanism through 
which we can orchestrate our efforts.^' 



On-site Visits 



Is It absolutely essential that on-site visits be made to projects being 
considered for validation? JORP makes all decisions without on-site visits, 
and witti only 10 pages of evidence of a program's effectiveness. ^Although the 
panel Is made up of 22 hl^ly trained specialists, four of whom must agree 
that a program or practice Is •effective If It Is to meet with JDRP approval. 



the panel has met with criticism because of Its failure to Include a visit to 
the site of each program or practice It screens. The NASAC-IVD report 
recommends the JDRP Incorporate a site visit Into Its procedures.*^ 

It would be Impossible for every state with a validation program to locate a 
group of evaluation specialists who would be qualified to evaluate the data 
from all programs and practices submitted to the state for validation. States- 
would a1$oJiayJ-to maintain panels of content specialists and experts able to 
validate the exportabllity of a program or practice. The possibility of a 
regional panel established by several cooperating states might be a feasible, 
though untested, alternative. 

The use of on-site teams means that apparently weak data or inconsistent 
evidence, a frequent problem for local projects, has a second chance. An 
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evaluation specialist working with project staff on'slte can- assist the staff 
In presenting the data In a different way or can help them clarify the 
evidence. For those programs or practices which are not ready for validation, 
an on-/lte team will be able to offer guidance for future attempts at 
validation. ' 



The In^ortance cf the on-site visit was an Issue -discussed at the six-state 



validation conference sponsored by the Southwest Educational Development 
Laboratory's Regional Exchange Project In April 1979. 

The temptation to validate a program based only on developer-writer 
descriptions plus examination of program products should be resisted. Ask 
whether the program, the process, or the product Is to be validated. The 
villidatton.. questions, which need to be answered will differ for each 

categbr;7.,-andJthe-usefu.lness of~the^aI1dat1on-data-wlU d 

accordingly. Paper files— collections of reports, brochures, teaching 
materials,, and the llksr-wl IT testify to procedures and processes. In 
general^ and will Indicate the quality of the teaching materials. What 
will not be clear Is the actual operation of the program in a school, and 
Its effects on teachers, students, and school routine. Paper files will 
not show whether Implementation Is occurring In all classrooms, or whether 
the manner of Implementation differs significantly across classrooms. 
So-called program'flles, on the other hand, address these questions 
because they will prcvlde for on-site visits either through a visiting 
validation team during the validation process, or bv the establishment of 
.demonstration sites during the dissmlnatlon stage.^^ 



A serious problem with on-site visits Is potential Inconsistency between 

'teams. Training for the IVO process Is conducted by personnel who have been 

trained by USOE. Although this strategy Is Intended to minimize the teams' 

differences In operation, IVO has drawn criticism because this 

decentralization procedure has led to site reviews which vary greatly In 
50 

stringency. States that do their own t)^a1n1ng might reduce this problem 
to some extent, but any time that more than one team is responsible for 
ecnducting the site visits in a state, the possibility of inconsistency among 
teams becomes a potential weakness in the procedure. 



The NASAC review of the IVO procedui^e led ,to a strong recommendation for 
continuance of on-site validation of programs and practices. The executive 
summary lists several potential solutions to the problems of high 

administrative costs of the program and the need for Increased state autonomy: 

1. The U.S. Office of Education [should] train persons designated by 
Federal -State authorities as validation on-site team leaders. The 
training \«ould be conducted In accordance.ywith the national 
validation process and within Joint Federal-State ^guidelines. An 

^H[itteiivt-^hou4<i-be-made-to-4r*ifWHie-persofVi>er^t^^ 

recommended that at least two people be trained as team leaders for 
each state. (State Facilitators— with existing NDN and state-agency 
linkages— are among the types of Individuals seen as being 
appropriate team leaders*) 

2. The USOE-tralned team leaders would have the resonslb.lHty to work 
with the Chief State School Officer within his or her state, or a 
person designated by the CSSO, to acquaint the SEA with the new 
validation process. The goal of this task would be to either gain 

_ SEA acce ptance of the Joint-agency procesSt or to assist In the 
modification of a State-sponsored process to correlate as closely as 
possible with USOE-approved procedures. 

3. The team leaders would have the responsibility. In cooperation with 
SEA officials, for coordinating the validation of submitted 
projects. Included In this responsibility would be the charge of 
org^anlzlng and training team members from within the state, and for 
conducting the on-site visitation Itself. Moreover, It Is 
recommended that small numbers of trained persons be maintained 
within each state to ensure the availability of well-quall/ied staff 
for each. specif 1c area of the validation process and to promote 
consistency of judgments. 

4. Should an out-of-state team member be selected for participation In 
an on-site validation, funds should be provided within the grant 
awarded to the coordinating agency to facilitate coverage of such 
costs as would accrue to the agency. (For example. If a State 
Facilitator Project Is selected to serve as the USOE-deslgnated team 
leader/trainer, funds would be Included In the NON's grant award to 
the Project to cover costs of out-of-state validators.) 

f^. In order to promote State autonomy, or SEA options In the on-site 
validation process, the Chief State School Officer should have the 
choice of trained team leaders within his or her state or region. 
The CSSO would have the authority to select preferred team leaders to 
coordinate all validation on-site activities within the State. 51 
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other Issues 

Several additional issues surface when the literature on validation is 
examined and when practitioners are asked about their concerns: 

1. It is important to keep in mind the teacher or the classroom as the 
- - unit for validation and not focus exclusively on administrative 
concerns, because itTis' the teacher who must know the essential , 

el e ni e jit s o f ~ttre~^qvatlPTn wderstawd why all or thDwin«nems «rr 

important; understand the purpose, use, and value of evidences of 
effectiveness and the teacher's role In collecting the evidence; and 
determine the feasibility of the program in the classroom. * . 
(Participants in SEDL Validation Conference) 
„2. — There-must-be-a-di st inctlon-made between-the vari ous stages durl ng 
the life of a program: the development, the validation, and the 
dissemination stages. Too often, the distinction is lost and 
pressure .to validate a program still in the developmental stage 
results in completing neither of the first two stages 
satisfactorily. Emrick and Peterson, among others,-have concluded 
_ that the Implementation process takes three to five years, and only 
at that time can one begin to collect meaningful evaluation data. 
(SEDL Validation Conference) 

3. Should projects that are not state or federally funded be considered 

for validation? 

Programs based on "validation'* activities rule out 
non-mainstream programs which do noj: offer themselves well to 
evaluation. Some programs are easier to evaluate than others. 
Try to evaluate a physical fitness program which must rely on 
data which shows "hoW many students would have suffered from X 
but didn't because of this program." Programs that started out 
as WD projects are traditionally easier to validate than • . 

o 



ptrograms which have grown in. a district over a number of years 
without' a project base. Thus R&O projects are validated nrare 
often than home gmwn programs forcing tfie^nhovative teacher 
who does not have federal or state funding support out of the. 
Idea market pi ace. 52 

What are the state's responsibilities after validation? Among the 
possibilities are (a) setting up a repository of materials and/or 
descriptions of validated programs and practices; (b) producing and 
disseminating descriptive brochures; (c) providing funds for 
validated projects to develop brochures to answer requests for 

V 

Information; (d) sponsoring awareness conferences and educational 
fairs; (e) funding replication/adaptation grants; and (f) offering 
technical assistance to LEAs rep^icatihg/adapting programs or 
practicesT 

What should bs the responsi bill ties of the project staff of validated 
programs? These typically include (a) answering requests; (b) 
printing and distributing a fact sheet or brochure describing the 
program; (c) scheduling and handling visitors; and (d) participating 
In educational fairs. Some project staff also become very involved 
In helping other LEA's replicate/adapt their program. NDN Developer/ 
Demonstrators, of course, receive funds to i.-ticipate in such 
activities; most state validated projects o ft. An issue discussed 
by participants in a May 1979 seminar o- validation sponsored the 
.Research for Better Schools Regional Exchange Project was the 
question of how to provide incentives for program developers to 
facilitate effective dissemination. 
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6. There are a number of additional problems associated with cost. 
Tumbull outlines some of the difficulties which need to be addressed: 

. . . when a program has been developed at one site. It Is very 
difficult to specify the Initial costs and the recurring costs 
that another site would experience. Besides separating these 
sorts of costs, there ere severalother problems: LEAs 
experience regional-differences In salary- scales and other 
. I, ' resource costs; schools start with different existing resources; 
local accounting practices may be Inaccurate or at least"h1ghly 

Individualistic; and different configura tions of a program may 

result In different per-pupll costs. . ,^tt 

Another cost associated problem ^discussed by participants In the 

* RBS-sponsored validation seminar was how to help developers continue 

effective programs after federal funding Is withdrawn. This 'is 

especially a concern for Title IV-C funded projects, which have a 

- five-year -llmlVatlononfundlngv - ^ 

7. Should second generation programs— programs which result from an 
adoption or replication— be validated and/or disseminated? Second 
generation program directors, while possibly not aware of the details 
of a program's creation, may be more ob^^ectlve about program effects 
than the developers." Adopting sites may have a clearer Insight into 
Implementation problems and needs and may know more about matching a 
school's needs with an already developed program rather than a 
developing one. Potential adopters may value those Insights as much 
as program details. And,, of course. It Is those second generation 
sites that "prove* whether the program can be transported, and 
therefore validated, and therefore disseminated! (SEDL Validation 
Conference) 
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1 4. Shirlng Educational Success; A Handbook for Validation of Educational 

PractlcaSi First Edltioft . (Washington. p.C; USOE, National Advisory 
* council on-Supplementary Centers and Services,' National Association of 
State Advisory Council Chairmen^ and the State Departments of Education, 
iiwd*):, p« V« - 

2. G.^Kaistch Tallmadge, The Joint Dissemination Review Panel Ideabook 
(Washington, O.C.: USOE and ^NIE, .1977). : 

3. Ideabook . p. 2. . 

4^ fTdeabook, pp. 9ff. ' , • ' 
5. Ideabook. pp. 74-75. 

6.. llnda^ Reed, Ed.RatrIck, and David Holdzkom, Survey of State Procedures for 
the^Val Idati oh of Educational Programs . developed by the Research and 
Development Exchange, 1981. See this report for a discussion of state 
vlldati oh practic es. 

7s, Wllllii Hinze, Sharing Educational Success. v A Handbook for Validation of 
Educational Practices] Revised Edition (Washington. D.C: Association of 
, State Advisory councils and^ State Departments of Education with assistance 
from the U.S.. Office of Education [1979]), p. 2. 

8. H^nze, pp» 6-7. 

9. Hinze, p. 20. 
10. Hinze, pp. 26-27. 
11.. Hinze, pp. 9-11.. 

12. Hinze, p. 19-27. 

13. Hinze, p. 9. 

14. Ray E. Foster, Survey of Certification Methods Applied to Innovative 
Public School Prooranis (Tallahassee. Florida; Educational Innovations 

. section, Florida Department of Education, 1978). 

15. Foster, p. 3. 

16. Foster, p. 4. 

17. Foster, p. 6. 

18. Reed et al.. Executive Summary, pp. 3-4. 



19. Hinzt, p. 6. 

20. Association of School Business Officials (ASBO), Research Corporation.' A , 
Handbook for Identification. Validation, and Dissemination of Scho ol * " 
Business practices (Chicago; Illinois: ASBO. lyyg). oo. 19-gO. ^ 

21. ASBO, pp. 22-36^ 

22. G. Kasten Tallmadge and Donald P. Horst, A Procedural Guide for Validating 
Achleveweiit Gains In Educational Projects . Revised edition. Number 2 In a 

. Ssrles of Monographs on Evaluation in Education (Washington, O.C.: U.S. 
Government Printing Office, 1976), p. 1. 

23i A Procedura^l Guide. . , 

.24. Donald P. Horst, 6. Kasten Tallmadge, and Christine T. Wood, A Practical 
Guide to Measuring Project Impact on Student Achievement (Washington, 
D.C.-r , USOE, I9/S7. " 

25. A Procedural Guldr . p. 52. 

26. A Procedural Guide , pp. 51-54. 

27. A Procedural Guide , pp. 46-47. 

28. A Procedyral Guide , pp. 13-50. 

29. For Information about the School Practices Information Service, contact 
Deborah Pletro, Education Service Group, Inc., Field Office, Box 227, 

j, Weeks Mills, Maine 04361. 

30. John E. Bowers et al.. Identifying. Validating, and Multimedia Packaging 
of Effective Reading Programs . Final Report. (Palo Alto. California: 
/wencan institutes tor Research in the Behavioral Sciences, 1974), p. 8., 

31. Effective Reading Programs: Summaries of 222 Selected' Programs (Urbana, 
Illinois: Right to Read and the ERIC Clearinghouse on Reading and 
Comiunlcatlon Skills; distributed by the National Council of Teachers of 
English, 1975). 

32. Jarvis S. Phillips and Williain 0. Chappelle, Catalog of Promising 
Educational Programs and Practices. 1972-73 (Washington/ D.C.: Automated 
services, inc., n.a.). 

33. Catalog of Promising Educational Programs , pp. 1-11. 

34. John A. Emrick and Susan M. Peterson, A Synthesis of Findings Across Five 
Recent Studies of Educational 01 ssemination and.Change . Executive Summary 
ISan Francisco, California: Far West Laboratory for Educational Research 
and Development, 1978), p. 5. 
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35. Paper written by Lois Ellin-Oata for discussion of results of the Rand 
Study (^ember 1978), pp.. 26-27. » 

.«■.*.■--. 

36. Paul Beman, Peter VT. Greenwood, Mllbrey Wall In McLaughlin, and John 
Pincus. Fyteral ProqraiiS Supporting Educational Ci<ange. Vol. IV 
(Abridged) i A suyiary of the Findings in Revl«v (Santa Monica. 
California:- Tne Rand corporation, p. 4. 
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tducatlonal Change. Vol. VIII: Implementing and sustaining innovations 

. (Santa Monica, canfomla: Tlte Rand corporation, 1978K p^^ 16. 
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39. Michael Fullan and Alan Pomfret, "Research on Curriculum and Instruction 
biplementatlon," Review of Educational Research . 47 (Winter 1977): 
39lf392. ' - • 

40» "The IVO Process— Myths and Facts Relevant to Prospective Policy 
Directions for Exesplary'lducatlonal Programs Replication," Executive 
( Suimary (Portsmouth, New Hampshire: National Association of State 
Advisory Councils, 1979), -pp. 5-6. 

41. telephone conversa1;Ton1ijiW Seymour Rubair,~MvcR"2?7^1W9^ ~ 

42. Ideabook . pp. 61-72. 

43; "The IVO Process,'' p. 5. , 

44. Brenda Tumbull, "Exemplary Practice: A Report on Recent Searches" 
(Prepared for the NIE Conference on Exemplary Program and Practice, August 
1975),, p. 9. , 

45. Increasing the Use of Promising Practices information by Local Educational 
Agencies (Report on an NIE conference conducted by Contemporary Research 
Incorporated, September 1974); p. 21i. • 

46. 'Gregory M. Benson, Jr., "A Survey to Determine the Ext'ent of State 
Education Agency Efforts to Develop Data Bases af • SEA->-Funded Projects 
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ORIGINAL IVO SimrSSItM RCQMHtQCNTS 



AlttEVIATQ} FORK 



Pm I— Informtlmi «n4 Projtet Otscriptlen 



A« WIOJgCT IWFOfWTtOW (Including vtpvidlturts) 



I. mcr oesainton or FMjgcT 
GoaU and objtetlvts 

Coitttxts csMunltyt sctoolt student chtfictAHstlei « 
Proms dMcnlptlon: gndt Itvtl(s), ytars of optntlont sanpit sizt, curricula, MttrlaU, staffing, 
facllltiM, t1» Involvid, partntal 1nvo1v«iont, prasarvlct/lnstrvlct training, ate* 
Costs: touU par pupil, Initial lnvlaaantatlon, ongoing aalntananca, ate* 



c. EviDOcg Of gmcnvpics 

'>$unar1st in appropHata daull tHa avaluatlon avldanet fo^ tba affactivoiitss of tfit progrw or aodal 
In 9imtitm. yt ordar to bo acctptaPIo, tht avaluatlon naod not bo a strict axperliNntai dasign 
(l.a*, longitudinal aaasurts, rtndoo uslgnaant to traatpant and control^groups, ate*), although this 
typo of avaluatlon avidanco Mould ba tho «ost das irablo* Howavar, In ordor for a projaet to bo 
valldatod and ran— oniiiil for wido scala adoption thort mat ba soao kind of >1gh quality* objoctlva, 
•othodologlcilly sound, ouaotltatlvo asstsMant vhlch^daionstratu that tho projact In quostlon Is 
affactlvt and suporlor to othor aoro conaonlv usad approachts or aathods* Thus, In ordar to approvo 
tho dlsaoirinatlon of any projoets, tho Pam rtquira a dotallod tuMary of tho ralovant aridanco 
Includfng such things u: ^ 

• td» canduc to d tha avaluatlon; 

• saipla sizas; 

• IflprmMits or gains In Mhatavor outcoM naasuros woro anployad; 

• tha stttlstlal rollahlllty and aducatlonal-slgnlflcanca of thoso tuprovoMnts; 

• SOBO arldonco^t tho laprovMnts can bo attrlbutod to tho progna and aro not just "nonaal* or 
•na tural * gains (l.a., control group or nons coaparisons, or soaa astlaata of Mhat mid havo 
occurrad la tha ahsanca of tho progrsM)* 



0. gviflChct or ttsduacE spgcmcATiows 

SuMrlza In approprlata dotall tha avidancc prasantad for start-up, oporatlon and aanagaaant rtsourcos 
and costs* 



t. EVIDCWCE OP PtftJfCT nPOPTAaiim 

Swarlxo In appnspHato datall tho proctssas roqulrad for adoption of tht projact by anothar school 
district Including aatarlalt aqulpaant and staff training. 

P. LOOILY DCl^ PWtlCATIOhS AhO mTCRIAlS 

list locally dovalopod publlatlons and aatarlals avallabia for dissaalnatlon. Slv<3 titio, dascrlption 
and cast* 



UaWTCIPATED OUTCqCS W SPIWOPP FlhOIHO 

Swarlza by listing of unantldpatad outcoaas and spinoff findings. 
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MKT II**eff«ct1vtftMs/Suecaf $ 



A6MCVIATE0 FOHM 



^•ct o6J«ct1vt(s) 1dint1f1«d fdr vtlltetlon havt bttn , 
•miw>4 ptrforwtiKt of tftt Itirw h«s bttn Inprdvtd^ 



jprtctlontt EvI^MKt on offoctlvmu/aucctts Is to bt rtporttd by objtetlvt. Part II hu IS 1t«K. 
wtpond to ^II.IL!^ ^ mwinatod obitctiv> Yow will notd «s my sots of Ptrt II bUnk foras 
u you Mvo oojoctivos* 

only Mjor yfcioetlVM sbpultf fct prt^ttd for villditlon. Ust thtst objoctlvts btlow* NMiibor 
»woaitivtiy. TIM handbook conui4% only oHt sot of fonw. Ri^roduct u my tots of fonw u you 



A ■•JOT objoctlvt ropro s ow M a contrti or pnairy goal of tbo ^Jict which will bo rtflocttd In tha 
aMunt of affort. projoct suff tlaa, funds axpondod and anticipated oohavlor outeoni. Ordlnartly tht 
Mjor^octlm tdll bo llaltod to tM or throo for oach project and will rtflKt studont loamor 

t^Stato oach aaior objoctlvt to bo valldatad on a soparato sat of Part II, Effoctivantss/Succtsst fdms. 
Itabor oach objoctlvo for identification. Froa hort on. Identify the objective by Its usloned raaber. 
You will need u atny sets of Part II fones as there are objectives to be validated. 

Objectives should be sated In Mosurable tern reflecting who, upon ^::;»p1etion of ttie project 
intervention, (treetiwit), will be able to do whet at what levi? ^ornancn. Any objective not 
MOtlnf the necessary specifications of aeasurablllty will be ellsrinated frm furtimr review, 
investigation and validation. 



2« 



Provide evidence to Justify the need for the objective under consideration by describing the needs 
usessKent procedures and the related findings. 

A direct rtlatlonship should exist between the Identified needs and Vm anticipated behavioral change 
defined by the «Jor project objectives. 

3. Irlefly descHbe the intervention process for this objective including where approprtate the following 
elawents: grade levtl(s), maiber of participants, peHod of operation. curHcula, spedel mateHals, 
sttffing, facilities, scheduling pattern, parental comnlty Involvenant, preservlce/lnservlce 
trelning, etc. 

4(a). Provide the eveloatlfln design (pre<*post test, baseline data, post test only, axpeHaental control, 
etc.), which, «Aan mvlMnted, will provide the infomatlon necessary to detamlne to what extent 
the objective was aetually attained. 

4(b). For the objective under consideration, describe data collection procedures specifying type of data 

collected, Method of .dats collection, timelines, responsible persons, ate. Chart fonMt Is suggested. 

4(c). If evaluation infprsBtlon was collected on a sampling of the project's participants, descHbe 
saepllng technique. 6lve saapling siza and evidence of represenutlveness. 

If sampling procedure was not used, write *llot Applicable* (HA) below. 

4(d). Prorlda supporting evidence that the* attalment of the objectlt-e can be attributed to the project 
Intervention (treataent) activities. 

4(e). Where control groups exist, describe how they ««re selected and give sow Indication of their 
equivalency to the project group. 

If control groups were not used, Hrlte "Not Applicable" (NA) below. 

4(f). Identify and descHbe each Instruaant utilized In the evaluation. Instnnents can be standardized 
or locally developed tests, questionnaires, interview form, ratino foms, inventoHes, etc. 
Include InfPmtlon concerning the instrumt's validity, reliability. «nd sensitivity to Mssure 
tUs range, scope, and nature of the behavior neasured. 
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4(1). 



S!j?J2l^iJ'*L5T?L'^'?7il5!*-^ collection («dKlfi1strat1on of UtU, 1nv«)tor1tf. 
spodil tra1fi1fi9 or proet^rts. so 1nd1c«tt« 

«M to dotoct^ corrott trrors in data ■anagwant. _ 



SUSl^ fI2J12^^^Lfr?^!2i'i ^? tmtwnt and Inttrprvtatlon. 
ptrsow or afandot rttpOMlbla for data analysis. 



Includo ni 



of 



^ 2ttbl1sil3*fd^^St*2K ^ objoctlvt ^ attalntd at or abovt tht crttaHon ItvtKs} 

^' rSportad^fliiSiiS*^*^'^ statistical ovldanet provldt ovidtfict of tducatlona' signlflcanca of tht 

7. Stata tlM conclusions Wiich utra drawi frm tho rtsults roporttd. 



fm III*«4)asouret Spoclfl cations 



Msoyrct Spocifidtions: Sufflctont Informtlon Is provldod conctml^" 
luff!* «tart«<io, opyatlonal and aanaooMnt rpsourets In kind and amnt 
f^'!SflJS^^J!^J!\V'l'i^ iff*ct1v«ws/SuccMs. and Part IV. 
fr ytay ilty datr, wfll halp an Inttrtsttd sctool distrtct snka an 
In^omod decision ataut adoption/adaption of tht projtct practlct. 



Rtteuret by Typt and Ammt 



Objtctlvts addrttstd by tht prtetlct: 
1. . 
3. . 



unit of Iitttrvtntlon 



1/ 


STAXT^OP COSTS 


mMBD€XT COSTS 


OPEMTIONAl COSTS 


ITEM 


mooucn 




PKCOUCER 


AOCFTER 


PROOUC£R 


AOOPTEA 


• 

Atsotret listing by" 
Mtrti Codt (stt 
pagt 40; ust only 
thest bydgtt ItM 
appropHatt to tht 
specific practlct). 


s 


s 


$ 


$ 


S 


$ 
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Pm III— fttjouret SptclflCAtlons by CAttQory 

Olrtctlpns; If Vm prwtici can bt idepttd In pt'rts ttit ptrtj shouW bt 1dtnt1f1«d uid (UU provia«d for-tach* 
1* ^i^<nt data on eoits of th# fanowHw; 



Staff Davtlopwwt 
MUrlaU 
Faellltits 
Contractad Sanrlcas 
CqulpMit (rtntal. purehasa) 
f. Tmal 

9* OtHfr {daserlbt Mow) 
Tout cast of Xfm practica 



b. 
c« 

a* 



1« 61va total m wba r of laamars upon 
Mlildi costs Mara basod* 

J* Olvlda 1 lata h and doHva par laamar 
eoatr for aach cotum. 



Start-up 
Costs , 


Nanigaaant 

Costs 


OparatlOfiat 
Costs 









pm nr-Exportablllty 



A validatad practica Is axportabia If (1) It Is faulblo to transport 
It to otHor school districts and (2) It can ba adoptad or adaptad by 
attiar iciiool districts with sisllar noads and anvlronwits* Usad 
Intordiai^ly tHtli portabia, rapllabla. and coMaunlcabla. Howavar. 
la t»1s soctlon q^stlons U13 win ba Jud^ad In tanas of caiiplatanass 
of raaponso* 



Provlda a dascrlptlon of tha naad fdr^ this projact In othar d1stHcts > 

Is datamlna tha potantlal msrtar of adoptars for your progrsw/ You »1ght Includa Intarast alraady 



(Tha pttrposa of this quastlon 



shoiai In your projact, rasults of Stata naads assassoanu, ate.) If tha t^rojaet can ba adoptad In 
part, tha docuaantatloa of avidanca should substantlata this for aach app11c!b1a eoHponant. 



2. Will tha laamar actlvltlas ba cantl^ with Stata or local funds? {Othar than ESEA Titla III) 

3. If tha proiact Is valldatod, and fundad for dissaolnatlon. Is tt^ Soard of education wllllno to oparata 
^ aroiact as a dawwstratlon sita (l.o., aceapt tha rola as a ptoducar school. Saa daf1n1t1on.)« 

• ^ population (a.g., aoa. athnic coaposltlon. Inccaa laval, 

taachar axparlanca, faslly, urban/niral ) . 

SwMrIzo tha laamar Involvaaant In tha progna you faai nacassary for an adoptar to obtain s1«11ar 
rasulu* 

6. Oascrlbo tha natura of tha institutional varlablas (a.g.. tha school adolnlstratlon, taaching staff, 
physical fKintlaa) which you faal Mould ba critical to tha succass of tha projact In an adopting 
district* 



Oaacrlba any cownlty and ho«a Involvanwit cHtlcal to tha succass of tha projact (a.g., tha nacHSlty 
for paranul and conunlty participation, ate.). r - x » . / 



ERLC 



59 



64 



r 



V 



AmcviATED rm 

8« OtscHbt ciMrly 9M prttlstly tht «ct1v1t1ts cHtleal to Vm succtts of tht projoet. Tbm should bt 
tta projtct tiMRts of yw^ progrwjilileh would to N Inmllod In anothtr dIstHet f or ^ to 
roeofiHzt ttm progrtw as an adoption. If jfour pr^Joct c^n #l5o ^ adopttd In part basod on tht saparatt 
obJtctlytt. cluittr jfoiir rtsponsas by objtetlvt. Br'^trl^ dlscL^js tho pros and cons of partial adoption 
vs. run adoption of tho prognM. 



9. 



10. 



list utontlal Mttrlals (softMort) usod by studonts, toachtrs* and othirs and tho sourco* Cost 
fl9ur«t fdr itav llstod should bo Includod In tho Rtsouret Spoelfleatlons Soctlon. List tho sourco 
of tut aatirlaU; ftoirltif thoM In torn of tht prtvlously dottnirfnod critical prograa tloaants. 



Ooocrlbo any ttsantlal o^lpMot (hardMrt) and/or unlqut facllltlos roqolrod for tht projtct for 
adoption and Its ust by actlvltlct. Cost figurts for Ittw llsttd should bt Includod in tht 
fttsourct Sptclflcatlon ^tctlon. 

/ 

11. OtscrIN tht typts,, nuMrs and qualifications of ptrsonntl rtqulrtd to optratt tht projtct 
succatsiV.iy. Vltn this In ttm of nhat tnothtr district will nttd to adopt your prograa. 

12. Otscrlbt prooDdurts and Mtorlals ntctssary for ptrsonntl training and ttchnlcal assistanci^ rtqulrtd 
for Ino^cailatlon of dia program In anothtr sotting. 



13. Idantify sptc1*:il probltat (unlquo to this kind of projtct) which tht adopting dIstHct Might tncounttr 
In lipltMntlM tht projtct and dtscrlbt solutions. Excludt probltn which will not havt a btaHng on 
tht txpo^lllty of tht projttt* 



ERLC 



60 



65 



^ The RDx Collaborative Effort on the Validation of Educational Prctjrams and 
Practices includes four products: 

Survey of S tate Procedures for the Validation of Educational Programs , by 
Linda Reed, Ed Patrick, and David Holdzkom. St. Louis, Missouri: CSMREL, Inc., 
foi* the R4D Exchange, 1981. 

Survey of State Procedures for the Validation of Educational Programs. 
Executive Stinmary. bv Linda Reed. Ed Patrirk. anH n;.u<H Hnl^^trnm^ c^- • lquIs,. 
nlssourl: CEMREL, Inc., for the R4D Exchange, 1981. 

The Search for Quality Control in Dissemination of Educational Products and 

Practices: A Look at the Literature and Ma.lor issues , by Linda Reed. ~ 

St. Louis, Missouri: CEMREL, Inc., rsd Interpretation Service, 1981. 

Validation of Educational Programs. Practices and Productj: An Annotated 
SiFdoqraphy . prepared bv Karen Taimen. Mary Ann Ts;.apg, and ^anH«»a p.H'^r. 
St. Louis, Missouri: CEMREL, Inc., for the R&D Exchange, 1981. 
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